Storage system, method of controlling storage system, and storage device

ABSTRACT

The present invention provides a storage system and a method of controlling the storage system, in which a second site rapidly resumes system process when a first site is damaged. The storage system comprises a first site including a first storage device, a second site including a second storage device, and a third site including a third storage device, and the method of controlling the storage system comprises a step of making a logical volume of the second storage device consistent with a logical volume of the first storage device by remotely copying only the differential data between the logical volume of the first storage device and the logical volume of the second storage device from a logical volume of the third storage device to the logical volume of the second storage device when the first site is damaged.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application relates to and claims priority from Japanese PatentApplication No. 2003-309194, filed on Sep. 1, 2003, the entiredisclosure of which is incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a storage system and a method ofcontrolling the same.

2. Description of the Related Art

Disaster recovery is important in an information-processing system. As atechnique for recovery from a disaster, the standard technique is knownto store and management a copy of the data on a storage device at aprimary site by a storage device located at a remote site (hereinafter,this technique is referred to as ‘remote copy’). According to thistechnique, when the primary site is damaged, processes performed at theprimary site are continuously performed at the remote site by using datain the storage device located at the remote site.

In the above-described method, in order to continue to perform theprocess that was being performed at the primary site at the remote sitewhen the primary site is damaged, it is necessary to perform a remotecopy from the primary site to the remote site in real time. However, theprimary site and the remote site are often separated distantly from eachother. Therefore, if the remote copy from the primary site to the remotesite is performed in real time, it takes much time for datacommunication, and the processing performance of the primary site isdecreased. In an information-processing system required for highavailability (HA), the processing performance of the primary site shouldnot be decreased, and the process should be rapidly resumed at theremote site when the primary site is damaged.

SUMMARY OF THE INVENTION

The present invention is designed to solve the aforementioned problem,and it is an object of the present invention to provide a storage systemand a method of controlling the same.

To achieve this object, there is provided a method of controlling astorage system comprising a first storage device having a first storagevolume provided at a first site, a second storage device having a secondstorage volume provided at a second site, and a third storage devicehaving a third storage volume provided at a third site, the storagedevices being connected so as to communicate with each other, whereinthe method includes the steps of: storing a copy of data stored in thefirst storage volume in the second storage volume at a first time;writing the copy of data written in the first storage volume into thethird storage volume; storing, in the third storage device, a writehistory of the data written in the first storage volume as a firstdifferential management table after the first time; and allowing thethird storage device to make the contents of the data stored in thesecond storage volume consistent with the contents of the data stored inthe first storage volume using the first differential management tableand the third storage volume of the third storage device.

Here, in an information-processing system in which a first site is aprimary site, a second site is a remote site, and a third site is alocal site provided in the vicinity of the first site, it becomespossible to make the second storage volume consistent with the firststorage volume by remotely copying only the differential data betweenthe first storage volume and the second storage volume from the thirdstorage volume to the second storage volume when the first site isdamaged.

Therefore, it is possible to provide a storage system and a method ofcontrolling the storage system having the above effects.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a schematic configuration of aninformation-processing system according to the present embodiment;

FIG. 2 is a diagram illustrating a configuration of a disk array deviceaccording to the present embodiment;

FIG. 3 is a diagram illustrating a configuration of aninformation-processing apparatus according to the present embodiment;

FIG. 4 is a diagram illustrating a configuration of a channel controlunit in a storage device according to the present embodiment;

FIG. 5 is a diagram illustrating a table stored in a shared memory in astorage device according to the present embodiment;

FIG. 6 is a diagram illustrating a pair management table according tothe present embodiment;

FIG. 7 is a diagram illustrating a differential management tableaccording to the present embodiment;

FIG. 8 is a diagram illustrating a change in the state of each storagedevice when a normal operation is performed at the first site in theinformation-processing system according to the present embodiment;

FIG. 9 is a flow diagram explaining a processing flow in a firstinformation-processing apparatus when a normal operation is performed atthe first site in the information-processing system according to thepresent embodiment;

FIGS. 10A and 10B are flow charts explaining a processing flow in athird storage device when a normal operation is performed at the firstsite in the information-processing system according to the presentembodiment; and

FIGS. 11A and 11B are diagrams illustrating an operation in which alogical volume of a second storage device is made to be consistent witha logical volume of the first storage device in order to shift a secondsite into use as the normal system in the information-processing systemaccording to the present embodiment.

DESCRIPTION OF THE EMBODIMENT

[Configuration of Information-Processing System]

FIG. 1 is a block diagram illustrating the entire configuration of aninformation-processing system including a storage system 100 accordingto the present embodiment.

The information-processing system comprises a storage device 10(hereinafter, referred to as ‘a first storage device’) provided at afirst site, an information-processing apparatus 11 (hereinafter,referred to as ‘a first information-processing apparatus’) accessing thefirst storage device, a storage device 20 (hereinafter, referred to as‘a second storage device’) provided at a second site, aninformation-processing apparatus 21 (hereinafter, referred to as ‘asecond information-processing apparatus’) accessing the second storagedevice, and a storage device 30 (hereinafter, ‘a third storage device’)provided at a third site. As a detailed example of each of the sites,this could be a computer facility operated by an organization, such as auniversity or corporation, or a data center in which an operation of aweb server on the Internet or an operation of ASP (Application ServiceProvider) is performed. The information-processing system is built fordisaster recovery in the event of an earthquake, fire, typhoon, flood,lightning, act of terrorism, and the like. This is described in detailin U.S. patent application Ser. No. 10/096,375. The U.S. patentapplication Ser. No. 10/096,375 is included herein by reference.

In this embodiment, the first site is the above-described primary site,the second site is the above-described remote site, and the third siteis the above-described local site.

The storage devices 10, 20, and 30 provided at the respective sites areconnected so as to communicate with each other via a first network 40.The first network 40 is, for example, the Gigabit Ethernet (a registeredtrademark), an asynchronous transfer mode (ATM), a public line, and thelike.

Each of the information-processing apparatuses 11 and 21 is a computercomprising a central processing unit (CPU) and a memory, such as apersonal computer (PC), a workstation, or a main frame. Each of theinformation-processing apparatuses 11 and 21 may be composed of aplurality of computers connected to each other. In each of theinformation-processing apparatuses 11 and 21, an operating system is inuse. Application software is operated through the operating system.

For example, the application software could provide a function of anautomatic teller system of a bank or a seat reservation system of anairplane. The information-processing apparatuses 11 and 21, and thestorage devices 10 and 20 in the first site and the second site areinterconnected by communication means (e.g., a communication line and anetwork), respectively. The communication means is, for example, a localarea network (LAN), a storage area network (SAN), an Internet SmallComputer System Interface (ISCSI), a fibre channel, an EnterpriseSystems Connection (ESCON) (a registered trademark), a Fibre Connection(FICON) (a registered trademark), and the like.

FIG. 2 shows the detailed configuration of a disk array device describedas one example of the first to third storage devices 10, 20, and 30.Further, the first to third storage devices 10, 20, and 30 may be, forexample, semiconductor storage devices, in addition to the disk arraydevices. The disk array device comprises a channel control unit 101, aremote communication interface 102, a disk control unit 103, a sharedmemory 104, a cache memory 105, a switching control unit 106, which iscomposed of, for example, a crossbar switch, for connecting the abovecomponents such that they can communicate with each other, a managementterminal 107, and a storage device 108.

The cache memory 105 is mainly used to temporarily store datacommunicated between the channel control unit 101 and the disk controlunit 103. For example, if data input/output commands received from theinformation-processing apparatuses 11 and 21 by the channel control unit101 are write commands, the channel control unit 101 writes write datareceived from the information-processing apparatuses 11 and 21 into thecache memory 105. Further, the disk control unit 103 reads out the writedata from the cache memory 105 and writes it into the storage device108.

The disk control unit 103 reads out data input/output (I/O) requestswhich are written into the shared memory 104 by the channel control unit101, and writes or reads data into or from the storage device 108according to commands (e.g., commands of SCSI (Small Computer SystemInterface) standards) set up in the data I/O requests. The disk controlunit 103 writes the data read out from the storage device 108 into thecache memory 105. Further, the disk control unit 103 transmits notice ofthe completion of data write or data read to the channel control unit101. The disk control unit 103 may have a function for controlling thestorage devices 108 in a RAID (Redundant Array of Inexpensive Disks)level (e.g., 0, 1, 5) defined in a so-called RAID manner.

The storage device 108 is, for example, a hard disk device. The storagedevice 108 may be integrated with the disk array device or may beindependently provided therefrom. A storage area provided by the storagedevice 108 at each site is managed using a logical volume 109, which isa volume logically set up in the storage area, as a unit. Writing orreading data into or from the storage device 108 can be performed bydesignating an identifier, which is given to the logical volume 109.

The management terminal 107 is a computer for maintaining and managingthe disk array device or the storage device 108. The change of softwareor parameters carried out in the channel control unit 101 or the diskcontrol unit 103 is performed by the instructions from the managementterminal 107. The management terminal 107 may be built in the disk arraydevice or may be separately provided therefrom.

The remote communication interface 102 is a communication interface (achannel extender) for data communication with other storage devices 10,20, and 30, and data duplicated by a remote copy, which will bedescribed later, is transmitted via the remote communication interface102. The remote communication interface 102 converts an interface (e.g.,an interface, such as a Fibre Channel, an ESCON (a registeredtrademark), or a FICON (a registered trademark)) of the channel controlunit 101 into a communication method for the first network 40. It allowsdata transmission between the storage devices 10, 20, and 30.

Further, the disk array device may be, for example, a device functioningas a NAS (Network Attached Storage) that is configured to accept datainput/output requests according to file name designation from theinformation-processing apparatuses 11 and 21 by a protocol, such as anNFS (Network File System), in addition to a device having theabove-stated configuration.

The shared memory 104 is accessible by both the channel control unit 101and the disk control unit 103. The shared memory is used to deliver datainput/output request commands and also stores management information forthe storage devices 10, 20, and 30 or the storage device 108. In thisembodiment, a consistency group management table 200, a pair managementtable 201, and a differential management table 202 are stored in theshared memory 104, as shown in FIG. 5.

FIG. 3 is a block diagram illustrating the configuration of theinformation-processing apparatuses 11 and 21.

The information-processing apparatuses 11 and 21 each comprise a CPU110, a memory 120, a port 130, a recording medium reading device 140, aninput device 150, and an output device 160.

The CPU 110 is in charge of the control of the entireinformation-processing apparatuses 11 and 21, and realizes variousfunctions by executing programs stored in the memory 120. The recordingmedium reading device 140 is a device for reading out programs or datarecorded on a recording medium 170. The read programs or data are storedin the memory 120. Thus, for example, it is possible to read a storagedevice management program 121 or an application program 122, which isrecorded on the recording medium 170, from the recording medium 170using the recording medium reading device 140 and to store it in thememory 120. As the recording medium 170, a flexible disk, a CD-ROM, asemiconductor memory, and the like may be used. The recording mediumreading device 140 may be built in the information-processingapparatuses 11 and 21 or may be separately provided therefrom. Forexample, the input device 150 is used for inputting data into theinformation-processing apparatuses 11 and 21 by an operator. As theinput device 150, for example, a keyboard, a mouse, and the like areused. The output device 160 is a device for outputting information tothe outside. As the output device 160, for example, a display, aprinter, and the like are used. The port 130 is a device forcommunicating with the storage devices 10, 20, and 30. Further, The port130 may be used for communication between the information-processingapparatuses 11 and 21. In this case, for example, the storage devicemanagement program 121 or the application program 122 may be receivedfrom the information-processing apparatuses 11 and 21 via the port 130and may be stored in the memory 120.

The storage device management program 121 is a program for performingthe copy management of the data stored in the logical volume 109 that isincluded in the storage device 108. Various commands for performing thecopy management are transmitted to the storage devices 10, 20, and 30 toallow the copy management operation to be performed.

The application program 122 is a program for making theinformation-processing apparatus 11 and 21 execute various functions.For example, there are a function for realizing the above-statedautomatic teller system of the bank, a function for realizing the seatreservation system of the airplane, and the like.

FIG. 4 is a block diagram showing the configuration of the channelcontrol unit 101.

The channel control unit 101 comprises a CPU 211, a cache memory 212, acontrol memory 213, a port 215, and a bus 216.

The CPU 211 is in charge of the control of the entire channel controlunit 101 and executes a control program 214 stored in the control memory213. The copy management of data according to the present embodiment isrealized as the control program 214 stored in the control memory 213 isexecuted. The cache memory 212 is a memory for temporarily storing data,commands, and the like, which are communicated with theinformation-processing apparatuses 11 and 21. The port 215 is acommunication interface for communicating with theinformation-processing apparatuses 11 and 21 or for communicating withother devices in the storage devices 10, 20, and 30. The bus 216connects these devices to each other.

[Pair Management Table]

A pair management table 201 is a copy management table of the logicalvolume 109 of each of the storage devices 10, 20, and 30. The pairmanagement table 201 comprises ‘type of pair’, ‘copy manner’, ‘copysource device’, ‘copy destination device’, ‘copy source volume’, ‘copydestination volume’, ‘paring status’, and ‘consistency group’ columns.

A pair refers to a combination of logical volumes 109 formed by twological volumes 109. Further, a case in which two logical volumes 109forming a pair are in the same storage devices 10, 20 and 30 is called a‘local pair’, and a case in which they are in different storage devices10, 20 and 30 is called a ‘remote pair’. Of the logical volumes 109forming a pair, one is a main logical volume, and the other is asecondary logical volume. It is possible to combine a plurality ofsecondary logical volumes with one main logical volume.

If the information-processing apparatuses 11 and 21 instruct the copysource storage devices 10, 20, and 30 to form remote pairs, each of thecopy source storage devices 10, 20, and 30 update the pair managementtable 202 of the shared memory 104 in each of the copy source storagedevices 10, 20, and 30. Thereafter, the copy source storage devices 10,20, and 30 instruct the copy destination storage devices 10, 20, and 30to form the remote pairs. The copy source storage devices 10, 20, and 30having received the instruction update the pair management tables 202 ofthe shared memories 104 in the copy destination storage devices 10, 20,and 30.

The column ‘type of pair’ in the pair management table 201 representswhether the pair is a local pair or a remote pair. The column ‘copymanner’ represents whether a remote copy manner is a synchronous manneror an asynchronous manner when the pair is the remote pair. Further, theremote copy and the manner thereof will be described later. The columns‘copy source device’ and ‘copy destination device’ represent the storagedevices 10, 20, and 30 of the copy source and the storage devices 10,20, and 30 of the copy destination when the pair is the remote pair. Thecolumn ‘copy source volume’ represents a main logical volume of thepair, and the column ‘copy destination volume’ represents a secondarylogical volume of the pair.

The column ‘state of pair’ represents the state of the pair. As thestate of the pair, there are ‘under pair’, ‘under split’, and ‘underresynchronization’.

In case of the ‘under pair’, data written into the main logical volumefrom the information-processing apparatuses 11 and 21 is reflected tothe secondary logical volume. Thus, it is possible to surely equalizethe contents stored in the main logical volume with the contents storedin the secondary logical volume by matching the main logical volume tothe secondary logical volume.

In case of the ‘under split’, even though data is written frominformation-processing apparatuses 11 and 21 into the main logicalvolume, the data is not reflected to the secondary logical volume. Thatis, the ‘under split’ state is a state in which the correspondencebetween the main logical volume and the secondary logical volume at the‘under pair’ is released. Thus, the identification between the contentsof the main logical volume and the contents of the secondary logicalvolume is not secured. However, since the contents of the secondarylogical volume are not updated in the ‘under split’ state, the data canbe backed up at this time.

The ‘under resynchronization’ is a state in the course of shifting fromthe ‘under split’ to the ‘under pair’. That is, in the ‘under split’,the data update performed on the main logical volume is being reflectedto the secondary logical volume. If the reflection is completed, thestate of the pair becomes the ‘under pair’. Further, shifting the pairfrom the state of the ‘under split’ to the state of the ‘under pair’ iscalled pair re-forming.

The formation of a pair, the split of a pair, and the re-formation of apair are performed by the operator inputting instructions from the inputdevice 150 to the information-processing apparatuses 11 and 21 in whichthe storage device management program 121 is executed. The instructionsinput by the operator are transmitted to the channel control units 101in the storage devices 10, 20, and 30. Then, the channel control unit101 executes the control program 214 to perform the formation of thepair or a change of the state of the pair according to the instructions.The channel control unit 101 performs the control of the logical volume109 according to the pairing status of the formed pair. For example, thechannel control unit 101 reflects the copy of the updated data of themain logical volume to the secondary logical volume with respect to thepair in the ‘under pair’ state.

The change of the pairing status by the channel control unit 101 issequentially performed on each pair. The reason is that, for example, ina state in which a pair is formed by combining a plurality of secondarylogical volumes with one main logical volume as described above, when aplurality of pairing conditions is simultaneously changed, managementfor the main logical volume is complicated.

Further, while the formation of a pair or the shift of a paring statusis initiated by the instructions received by information-processingapparatuses 11 and 21 from the operator, the shift of the paring statusmay be automatically performed, for example, when a predetermined timeelapses without instructions from the operator. In addition, theformation of a pair or the shift of a paring status may be performed byreceiving instructions from the information-processing apparatuses 11and 21 connected via the port 215.

In this embodiment, a pair is set up as shown in the pair managementtable of FIG. 6. That is, the logical volume 109 (hereinafter, referredto as ‘a first logical volume’) at the first site forms a local pair.Further, the secondary logical volume (hereinafter, referred to as ‘afirst secondary logical volume’) of the local pair of the first logicalvolume and the logical volume 109 (hereinafter, referred to as ‘a secondlogical volume’) at the second site form a remote pair. The remote pairof the first main logical volume and the logical volume 109(hereinafter, referred to as ‘a third logical volume’) at the third siteis always in the ‘under pair’ state, and the third logical volume isalways consistent with the first main logical volume by the remote copyin a synchronous manner, which will be described later.

Further, data backup from the first logical volume to the second logicalvolume is performed as follows. First, the first storage device 10shifts the local pair of the first logical volume to the ‘under split’by the instructions from the first information-processing apparatus 11.When split is completed, the first storage device 10 re-forms the remotepair of the first secondary logical volume and the second logical volumeaccording to the instructions from the first information-processingapparatus 11. Further, the first information-processing apparatus cancontinue to perform the process using a main logical volume(hereinafter, referred to as ‘a first main logical volume’) of the localpair of the first logical volume during the re-formation of the remotepair.

[Consistency Group]

The number of a consistency group (i.e., a pair group) to which the pairbelongs is written into the column ‘consistency group’ of the paringstatus management table 201. Here, a consistency group refers to a groupformed of pairs of a plurality of logical volumes 109, which iscontrolled so that the shift to the split state thereof issimultaneously made. That is, a change of the paring status with respectto the plurality of pairs is sequentially performed on each pair asdescribed above. However, a plurality of pairs belonging to the sameconsistency group is controlled such that the shift to the split statethereof is simultaneously performed (hereinafter, it is referred to assimultaneity of the shift to the split state).

For example, a case is considered in which data is written from theinformation-processing apparatuses 11 and 21 to the storage devices 10,20, and 30 while the storage devices 10, 20, and 30 sequentially changea plurality of pairs belonging to the same consistency group from the‘under pair’ to the ‘under split’. If the consistency group has not beenformed, the write data is not reflected to the secondary logical volumewhen the data writing is intended with respect to the main logicalvolume of the pair the shift to the split state of which is completed.On the contrary, if the data writing is intended with respect to themain logical volume of the pair the shift to the split state of which isnot completed, the write data is reflected to the secondary logicalvolume. However, when the consistency group has been formed, the writedata is not reflected to the secondary logical volume, regardless ofwhether the shift to the split state is completed. This is because thedata writing from the processing devices 11 and 21 to the storagedevices 10, 20, and 30 is called after the split for the pair of theconsistency group has been initiated (the release of correspondence isinitiated).

As described above, forming a consistency group with respect to aplurality of pairs is particularly effective when one data item isstored in the plurality of logical volumes 109, for example, when thewrite data is too large to be stored in one logical volume 109, and whenone file data item is controlled so that it is stored in the pluralityof logical volumes 109.

In addition, securing simultaneity for the shift of each pair to thesplit state using the consistency group is effective even when there isa data write request or a data read request from theinformation-processing apparatuses 11 and 21 to the secondary logicalvolume.

That is, the reason is that, when the consistency group has not beenformed, data writing or data reading can be performed with respect to asecondary logical volume of the pair the shift to the split state ofwhich is completed, but data writing or data reading is prohibited withrespect to a secondary logical volume of the pair the shift to the splitstate of which is not completed.

Further, a split for a pair of the consistency group may be performed bydesignating the tart time thereof. The start time of the split isinstructed by a command transmitted from the information-processingapparatuses 11 and 21.

[Remote Copy]

When the above-stated remote pairs are formed in the logical volumes 109of the first to third storage devices 10, 20, and 30, the copy sourcestorage devices 10, 20, and 30 transmit data copied by the remote copyto the copy destination storage devices 10, 20, and 30. Thecorresponding relationship between the main logical volumes of the copysource storage devices 10, 20, and 30 and the secondary logical volumesof the copy destination storage devices 10, 20, and 30 is set up in theabove-stated pair management table 201. When the information-processingapparatuses 11 and 21 write data to the main logical volumes of the copysource storage devices 10, 20, and 30 while the paring status is the‘under pair’, the copy source storage devices 10, 20, and 30 transmitthe copy of the data to the copy destination storage devices 10, 20, and30 via the first network 40. Then, the copy destination storage devices10, 20, and 30 write the received data to the secondary logical volumes.While the pairing status is ‘under split’, the copy source storagedevices 10, 20, and 30 do not transmit the copy of the data to the copydestination storage devices 10, 20, and 30 even when theinformation-processing apparatuses 11 and 21 write data to the mainlogical volumes of the copy source storage devices 10, 20, and 30. Thecopy source storage devices 10, 20, and 30 transmit the copy of the datato the copy destination storage devices 10, 20, and 30 at the timingwhen the paring status is shifted to the ‘under resynchronization’, andthen the copy destination storage devices 10, 20, and 30 write thereceived data to the secondary logical volumes. That is, the remote copyis performed to make the contents of the main logical volumes of thecopy source storage devices 10, 20, and 30 consistent with the contentsof the secondary logical volumes of the copy destination storage devices10, 20, and 30, wherein the copy source storage devices 10, 20, and 30correspond to the copy destination storage devices 10, 20, and 30.

Further, the remote copy manner includes a synchronous manner and anasynchronous manner, which are determined in the ‘copy manner’ column ofthe differential management table 202. In the case of the synchronousmanner, if the information-processing apparatuses 11 and 21 request thecopy source storage devices 10, 20, and 30 to write data into the mainlogical volumes, the copy source storage devices 10, 20, and 30 firstwrite data into the main logical volumes thereof. In addition, the copysource storage devices 10, 20, and 30 transmit the same data as thewrite data to copy destination storage devices 10, 20, and 30. Then, thecopy destination storage devices 10, 20, and 30 write the received datainto the secondary logical volumes and notify that fact to the copysource storage devices 10, 20, and 30. The copy source storage devices10, 20, and 30 that have received this notification give notice thatdata writing has been completed to the information-processingapparatuses 11 and 21.

In such a synchronous manner, the information-processing apparatuses 11and 21 have notice of the completion of data writing after it isconfirmed that data is written to both the main logical volumes andsecondary logical volumes. Accordingly, in the synchronous manner, thecorrespondence between the contents of the main logical volumes and thecontents of the secondary logical volumes is secured at the time whenthe information-processing apparatuses 11 and 21 receive notice of thecompletion. However, the information-processing apparatuses 11 and 21receive notice of the completion until data is completely written intothe secondary logical volume. Thus, in the synchronous manner, responsetime until the completion notice returns to the information-processingapparatuses 11 and 21 after a data write request is transmitted frominformation-processing apparatuses 11 and 21 to the copy source storagedevices 10, 20, and 30 is generally longer than that in the asynchronousmanner.

Meanwhile, in the asynchronous manner, when the information-processingapparatuses 11 and 21 request the copy source storage devices 10, 20,and 30 to write data into the main logical volumes, the copy sourcestorage devices 10, 20, and 30 write data into the main logical volumes.Further, the copy source storage devices 10, 20, and 30 transmit thesame data as the write data to the copy destination storage devices 10,20, and 30. The copy destination storage devices 10, 20, and 30 writethe received data into the secondary logical volumes and notify thatfact to the copy source storage devices 10, 20, and 30. Here, if thecopy source storage devices 10, 20, and 30 write data to the mainlogical volumes, they immediately transmit the completion notice to theinformation-processing apparatuses 11 and 21, irregardless of whetherdata is written to the secondary logical volumes of the copy destinationstorage devices 10, 20, and 30. Thus, in the asynchronous manner,response time with respect to the information-processing apparatuses 11and 21 is generally shorter as compared to the synchronous manner.However, the data coincidence between the main logical volumes and thesecondary logical volumes is not necessarily secured. Further, in thecase of the asynchronous manner, the copy source storage devices 10, 20,and 30 manage data that is not reflected to the secondary logicalvolumes of the copy destination storage devices 10, 20, and 30.

In this embodiment, a remote pair between the first main logical volumeand the third logical volume performs a remote copy in the synchronousmanner. Thus, the contents of the first main logical volume are alwaysconsistent with the contents of the third logical volume.

[Differential Management Table]

The differential management table 202 shows whether any block of acertain logical volume 109 is updated on the basis of a predeterminedtime. As shown in FIG. 5, the table is generally recorded in the sharedmemories 104 of the storage devices 10, 20, and 30 in a bitmap format,and each bit corresponds to a block in the logical volume 109. Thedifferential management table 202 in which an initial state of each itemis ‘0’ is recorded in the shared memories 104 of the storage devices 10,20, and 30. If the storage devices 10, 20, and 30 write data in thelogical volumes 109, each of them updates a bit on the differentialmanagement table, which indicates a block in which the data is stored,to ‘1’.

In this embodiment, whether the third storage device 30 updates anyblock of the third logical volume on the basis of the time when thelocal pair of the first logical volume is split is recorded in theshared memory 104 of the third storage device 30 as the differentialmanagement table 202. As described above, the remote pair between thefirst main logical volume and the third logical volume is always the‘under pair’, thereby performing the remote copy in the synchronousmanner. Thus, the differential management table 202 of the third storagedevice 30 indicates a write history of data recorded in the first mainlogical volume after the local pair of the first logical volume issplit. That is, in the differential management table 202, data in thethird logical volume of a block in which ‘1’ is set up is data recordedin the first main logical volume after the local pair of the firstlogical volume is split. In addition, the first information-processingapparatus 11 backs up the first secondary logical volume to the secondlogical volume in a state in which the local pair of the first logicalvolume is split.

In this way, when the first site is damaged, the third storage device 30makes the second logical volume consistent with the first main logicalvolume by remotely copying only data in the third logical volume, whichis set up to ‘1’ in the differential management table 202 of the thirdstorage device 30, to the second logical volume.

However, when the first site is damaged while the first storage device10 performs a remote copy from the first secondary logical volume to thesecond logical volume, the second logical volume has not been updated soas to be consistent with the first main logical volume at the time whenthe local pair of the first logical volume is split the last time. Thus,the third storage device 30 can make the second logical volumeconsistent with the first main logical volume only with the differentialmanagement table 202 at the time when the first logical volume is splitthe last time.

Thus, in this embodiment, the third storage device 30 records thedifferential management table 202 at the time when the local pair of thefirst logical volume is split the time before last and the last time.Accordingly, when the first site is damaged while the first storagedevice 11 performs a remote copy from the first secondary logical volumeto the second logical volume, the third storage device 30 can make thesecond logical volume consistent with the first main logical volumeusing the differential management table 202 recorded at the time whenthe local pair of the first logical volume is split the time beforelast.

FIG. 7 shows an example in which the first site is damaged while thefirst storage device 10 is performing a remote copy from the firstsecondary logical volume to the second logical volume. In thedifferential management table 202, ‘1’ is set up with respect to a blockof the third logical volume on which the third storage device 30performs update, and FIG. 7 shows the contents of write data in thelogical volume for the sake of convenience in the present invention. Asshown in FIG. 7, at the time when it is damaged, update is completed upto the data ‘5’ in the remote copy from the first secondary logicalvolume to the second logical volume, which is performed by the firststorage device 10. However, since the update of the subsequent data ‘9’is not completed, the second logical volume is not consistent with thefirst main logical volume at the time when the local pair of the firstlogical volume is split the last time. Thus, the third storage device 30remotely copies, to the second logical volume, data ‘1’, ‘5’, ‘9’, ‘3’,‘4’, and ‘2’ of the blocks 1 to 6 in which ‘1’ is set up in thedifferential management table at the time when the local pair of thefirst logical volume is split the time before last. Accordingly, thethird storage device 30 can make the second logical volume consistentwith the first main logical volume at the time when the first site isdamaged.

Further, if the storage devices 10, 20, and 30 receive differentialdelete requests from the information-processing apparatuses 11 and 21,they update all bits of the differential management tables 202 to ‘0’.The information-processing apparatuses 11 and 21 may perform thedifferential delete requests by designating a differential delete starttime. In addition, the information-processing apparatuses 11 and 21 caninstruct differential deletion in a consistency group unit when thelogical volume 109 that performs the update management in thedifferential management table 202 forms the consistency group.

In this embodiment, the third logical volume has two differentialmanagement tables 202, as described above. If the third storage device30 receives a differential delete request from the firstinformation-processing apparatus 11, it deletes only the differentialmanagement table 202 at the time when the local pair of the firstlogical volume is split the time before last, and begins to record onthe differential management table 202 update information at the timewhen the local pair is newly split.

Transition in General Operation

FIG. 8 illustrates the transition of a data state in each site at thetime T1 to T3 when the first information-processing apparatus 11operates a normal system. Further, STn written in the logical volume 109indicates that the logical volume 109 is consistent with the first mainlogical volume at the time Tn. Further, Δtxy indicates a differentialmanagement table 202 from the time Tx to the time Ty.

First, a state at the time T1 will be described. Assuming that the lastsplit time of a local pair in the first logical volume is T0, a state isillustrated in which the first information-processing apparatus 11splits the local pair of the first logical volume at the time T1. Thestate is as follows: the local pair of the first logical volume is the‘under split’, a remote pair of the first main logical volume and thethird logical volume is the ‘under pair’, and a remote pair of the firstsecondary logical volume and the second logical volume is the ‘undersplit’. The first information-processing apparatus 11 designates thetime T1, which is the same as the split of the first logical volume, andtransmits a delete request from the differential management table 202 tothe third storage device 30. The third storage device 30 having receivedthe request deletes, at the time T1, the differential management table202 recorded at the time when the local pair of the first logical volumeis split the time before last. Further, the second logical volume is inthe state of the first main logical volume at the time T0.

Next, a state at the time T2 will be described. In this state, the firststorage device 10 re-forms a local pair of the first secondary logicalvolume and the second logical volume according to an instruction fromthe first information-processing apparatus 11. The firstinformation-processing apparatus 11 monitors the pairing status of aremote pair of the first secondary logical volume and the second logicalvolume. Then, when the paring status becomes the ‘under pair’, the firstinformation-processing apparatus 11 instructs the first storage device10 to split the remote pair.

Finally, a state at the time T3 will be described. In this state, thefirst information-processing apparatus 11 instructs the first storagedevice 10 to re-form a local pair of the first logical volume. Then, thefirst storage device 10 re-forms the local pair of the first logicalvolume.

As describe above, the transition from the state at the time T1 to thestate at the time T3 repeatedly occurs when the first site is notdamaged. The operation of the first information-processing apparatus 11and the third storage device 30 during this transition will bedescribed.

First, the operation of the first information-processing apparatus 11will be described with reference to FIG. 9. The firstinformation-processing apparatus 11 instructs the first storage device10 to split the local pair of the first logical volume with timedesignation (S911). Here, the split is executed on the local pair in aconsistency group unit. Further, the first information-processingapparatus 11 transmits to the third storage device 30 a differentialdelete request in which the same time as the split is set up as adifferential delete start time (S912). Further, the differential deleterequest is performed on the remote pair in the consistency group, whichis the same as the local pair on which the split is performed. When thefirst information-processing apparatus 11 receives split completionnotice from the first storage device 10 and differential deletecompletion notice from the third storage device 30 (S914) after thesetup time has elapsed (S913), it instructs the first storage device 10to re-form the remote pair of the first secondary logical volume and thesecond logical volume (S915). The first information-processing apparatus11 monitors the pairing status of the remote pair of the first secondarylogical volume and the second logical volume. Then, when the state ofthe pair becomes the ‘under pair’ (S916), the firstinformation-processing apparatus 11 instructs the first storage deviceto split the pair (S917). Subsequently, the first information-processingapparatus 11 instructs the first storage device 10 to re-form the localpair of the first logical volume (S918). The firstinformation-processing apparatus 11 repeatedly executes such a series ofprocesses.

Next, the operation of the third storage device 30 will be describedwith reference to FIGS. 10A and 10B. When the third storage device 30receives a differential delete request from the firstinformation-processing apparatus 11, it sets up a differential deletestart time to a consistency group management table 200 (S1011). Thethird storage device 30 monitors a data write request from the firststorage device 10 during the differential delete start time (S1012).When the third storage device 30 receives the data write request fromthe first storage device 10, it confirms whether data is recorded in thelogical volume 109 in which the differential management table 202 hasnot yet been deleted (S1013). If the data writing is performed in thelogical volume 109 in which the differential management table 202 hasnot yet been deleted, the third storage device 30 compares the time setin the write request with the differential delete start time (S1014).When the time set in the write request is later than the differentialdelete start time, the third storage device 30 first deletes thedifferential management table 202 of a logical volume 109, which is awrite object (S1015). The third storage device 30 then records thewritten information on the differential management table 202 (S1016).Further, if a data write request from the first storage device 10 isdirected to the logical volume 109 in which the differential has alreadybeen deleted, the third storage device 30 does not perform the deletionof the differential management table 202 and records the writteninformation on the differential management table 202 (S1017). Further,even though the write is directed to the logical volume 109 in which thedifferential management table 202 is not deleted, the third storagedevice 30 does not perform the deletion of the differential managementtable 202 and records the written information on the differentialmanagement table 202 when the time set in the write request is fasterthan the differential delete start time (S1017).

As such, the third storage device 30 compares the time set up in thewrite request received from the first storage device 10 with thedifferential delete start time set up in the differential delete requestreceived from the first information-processing apparatus 11, andperforms the data writing on the differential management table 202 andthe data deletion from the differential management table 202 in a timesequence. That is, the data writing on the first main logical volume andthe split for the local pair of the first logical volume performed bythe first storage device 10, and the data writing on the differentialmanagement table 202 and the data deletion from the differentialmanagement table 202 performed by the third storage device 30 isperformed in regular order. Accordingly, it is secured that thedifferential management table 202 of the third storage device 30 is awrite history of data written to the first main logical volume after atime when the local pair of the first logical volume is split.

Further, if the third storage device 30 does not receive a write requestfrom the first storage device 10, it deletes the differential managementtable 202 of the logical volume 109 which has not yet been deleted(S1019) after the differential delete start time has been elapsed(S1018). Since the third storage device 30 deletes the differentialmanagement table 202 in a consistency group unit, it confirms whetherthe deletion of the differential management tables 202 for all pairs inthe consistency group is completed (S1020). If the deletion of thedifferential management table 202 for all pairs in the consistency groupis completed, the third storage device 30 deletes the differentialdelete start time from the consistency group management table (S1021)and transmits to the first information-processing apparatus 11 a noticethat the differential management table 202 is completely deleted(S1022). Further, each logical volume 109 has two differentialmanagement tables 202 as described above. Therefore, when the thirdstorage device 30 receives a differential delete request from the firstinformation-processing apparatus 11, it deletes only the differentialmanagement table 202 recorded at the time when the local pair of thefirst logical volume is split the time before last.

[Process when First Site is Damaged]

The operation of making the second logical volume consistent with thefirst main logical volume in order to operate the second site as a mainsystem when the first site is damaged will be discussed. First, thesecond information-processing apparatus 21 instructs the second storagedevice 20 to acquire the paring status of the remote pair of the secondlogical volume and the first secondary logical volume. The secondstorage device 20 having received this instruction refers to the pairmanagement table 202 in the shared memory 104 of the second storagedevice 20 and transmits the contents of the table to the secondinformation-processing apparatus 21.

Next, the second information-processing apparatus 21 instructs thesecond storage device 20 to form a remote pair in which a third logicalvolume is a main logical volume and a second logical volume is asecondary logical volume. Further, the second information-processingapparatus 21 transmits to the second storage device 20 whether theremote pair of the second logical volume and the first secondary logicalvolume was in the ‘resynchronization’ state.

When the second information-processing apparatus 21 instructs the secondstorage device 20 to form a pair of the third logical volume and thesecond logical volume, the second storage device 20 updates the pairmanagement table 202 of the second storage device 20. Further, thesecond storage device 20 transmits to the third storage device 30 thepair formation instruction and the state of the pair of the secondlogical volume and the first secondary logical volume received from thesecond information-processing apparatus 21. When the third storagedevice 30 receives them, it updates the pair management table 202 of thethird storage device 30. Then, the third storage device 30 performs aremote copy from the third logical volume to the second logical volumebased on the state of the pair of the second logical volume and thefirst secondary logical volume, which is received from the secondstorage device 20.

That is, if the state of the remote pair of the second logical volumeand the first secondary logical volume is not the ‘resynchronization’,the third storage device 30 refers to the differential management table202 at the time when the local pair of the first logical volume is splitthe last time. Further, if the state of the remote pair of the secondlogical volume and the first secondary logical volume is the ‘underresynchronization’, the third storage device 30 refers to thedifferential management table 202 at the time when the local pair of thefirst logical volume is split the time before last. The third storagedevice 30 remotely copies only the third logical volume block on thereferred differential management table 202, in which ‘1’ is set up, tothe second logical volume.

FIG. 11A illustrates an example in which the remote pair of the secondlogical volume and the first secondary logical volume is not in the‘resynchronization’ state. Assuming that the time when the local pair ofthe first logical volume is split the time before last is T0 and thetime when it is split in the last stage is T1, FIG. 11A illustrates asituation in which the first site is damaged at the time T3. Since thepairing status of the local pair of the second logical volume and thefirst secondary logical volume is the ‘under split’, the third storagedevice 30 remotely copies the block recorded on the differentialmanagement table 202 (ΔT13) at the time T1 from the third logical volumeto the second logical volume. Therefore, it is possible to equalize thesecond logical volume, which is ST3, with the first main logical volume.

Next, FIG. 11B illustrates an example in which the remote pair of thesecond logical volume and the first secondary logical volume is in the‘resynchronization’ state. Assuming that the time when the local pair ofthe first logical volume is split the time before last is T0 and thetime when it is split in the last stage is T1, FIG. 11B illustrates asituation in which the first site is damaged at the time T2. Since thestate of the local pair of the second logical volume and the firstsecondary logical volume is the ‘resynchronization’, the third storagedevice 30 remotely copies the block recorded on the differentialmanagement table 202 (ΔT02) at the time T0 from the third logical volumeto the second logical volume. Therefore, it is possible to equalize thesecond logical volume, which is ST2, with the first main logical volume.

In the aforementioned embodiments of the present invention, theembodiments are intended to easy understanding of the present invention,but the present invention is not limited thereto. The present inventionmay be modified without departing the spirit thereof, and includes itsequivalents.

1-29. (canceled)
 30. A storage system, comprising: a first disk array system, in a primary site, coupled to a host computer and having a plurality of first disk drive units in which data are stored and a first controller which controls to store data sent from said host computer to a primary volume configured by some of said first disk drive units, a second disk array system, in a remote site, coupled to said first disk array system and having a plurality of second disk drive units in which data are stored and a second controller which receives data of said primary volume from said first disk array system with an asynchronous remote copy procedure and controls to store data received from said first disk array system to a secondary volume configured by said second disk drive units, a third disk array system, in a local site, coupled to said first disk array system and having a plurality of third disk drive units in which data are stored and a third controller which receives data of said primary volume from said first disk array system with a synchronous copy procedure and controls to store data received from said first disk array system to a third volume configured by said third disk drive units, at least one storage area, in said first disk array system, having a first status in which said storage area is related to updates if said primary volume is updated by said host computer, and a second status in which said storage area is not related to updates if said primary volume is updated by said host computer; a first area, in said third disk array system, storing information related to data received several times from said first disk array system with said synchronous copy procedure, said data related to said information in said first area corresponding to said updates related to said storage area of said first status and a second area, in said third disk array system, storing information related to data received several times from said first disk array system with said synchronous copy procedure, said data related to said information in said second area corresponding to said updates during a time that said storage area is in said second status; wherein said third disk array system clears information stored in said first area after said second disk array system received data which relates to information stored in said first area, and wherein information stored in said first area and information stored in said second area are cleared at different times.
 31. A storage system according to claim 30, wherein: said synchronous copy procedure is a copy style that said first disk array system sends acknowledge to said host computer after said third disk array system receives first data sent from said first disk array system, if said first disk array system receives said first data sent from said host computer.
 32. A storage system according to claim 30, wherein: said asynchronous remote copy procedure is a copy style that said first disk array system sends acknowledge to said host computer without relation to that said third disk array system receives second data sent from said first disk array system, if said first disk array system receives said second data sent from said host computer.
 33. A storage system according to claim 30, wherein: said first area is a bitmap, and said second area is a bitmap.
 34. A storage system according to claim 30, wherein: said first area is a first bitmap, and said information stored in said first bitmap includes information of a position in which data received from said first disk array system are stored.
 35. A storage system according to claim 30, wherein: information stored in said first area are cleared periodically.
 36. A storage system according to claim 30, wherein: said first area and said second area are acted the rotation of the roles on a periodic basis.
 37. A storage system according to claim 30, wherein said first status and said second status of said storage region are acted the periodic rotation.
 38. A storage system according to claim 30, wherein: said data transfer from said first disk array system to said second disk array system with said asynchronous remote copy procedure is acted on a periodic basis.
 39. A storage system according to claim 30, wherein: said third disk array system sends data which relates to information stored in at least one of said first area and said second area to said second disk array system, if said first disk array system has a disaster.
 40. A storage system according to claim 30, wherein: said third disk array system clears said information stored in said first area based on one of change between said first status and said second status of said storage area, after said second disk array system received data which relates to information stored in said first area.
 41. A storage system, comprising: a first disk array system, in a primary site, coupled to a host computer and controlling to store data sent from said host computer to a primary volume, a second disk array system, in a remote site, coupled to said first disk array system and receiving data of said primary volume from said first disk array system with an asynchronous remote copy procedure and controlling to store data received from said first disk array system to a secondary volume, a third disk array system, in a local site, coupled to said first disk array system and receiving data of said primary volume from said first disk array system with a synchronous copy procedure and controlling to store data received from said first disk array system to a third volume, at least one storage area, in said first disk array system, having a first status in which said storage area is related updates if said primary volume is updated by said host computer, and a second status in which updates related to said storage area are transferred to said second disk array system; a first area, in said third disk array system, storing information related to data received several times from said first disk array system with said synchronous copy procedure during a first time period which is a time period that said storage area is in said first status; and a second area, in said third disk array system, storing information related to data received several times from said first disk array system with said synchronous copy procedure during a second time period which is a time period that said storage area is in said second status, said second time period succeeding said first time period, wherein said third disk array system clears information stored in said first area or said second area based on one of change between said first status and said second status of said storage area, after said second disk array system received data which relates to information stored in said first area or said second area, and wherein information stored in said first area and information stored in said second area are cleared at different times.
 42. A storage system according to claim 41, wherein: said several times related to information stored in said first area correspond to number of times of receiving write data from said host computer to said first disk array system during said first time period, and said several times related to information stored in said second area correspond to number of times of receiving write data from said host computer to said first disk array system during said second time period.
 43. A storage system according to claim 41, wherein: said first area is a first bitmap, and said cleared first bitmap is used to store information related to data received from said first disk array system with said synchronous copy procedure during a third time period which is a time period succeeding said second time period.
 44. A storage system, comprising: a first disk array system, in a primary site, coupled to a host computer and having a plurality of first disk drive units in data are stored and a first controller which controls to store data sent from said host computer to a primary volume configured by some of said first disk drive units, a second disk array system, in a remote site, coupled to said first disk array system and having a plurality of second disk drive units in which data are stored and a second controller which receives data of said primary volume from said first disk array system with an asynchronous remote copy procedure and controls to store data received from said first disk array system to a secondary volume configured by said second disk drive units, a third disk array system, in a local site, coupled to said first disk array system and having a plurality of third disk drive units in which data are stored and a third controller which receives data of said primary volume from said first disk array system with a synchronous copy procedure and controls to store data received from said first disk array system to third volume configured by said third disk drive units, at least one storage area, in said first disk array system, having a first status in which said storage area is related to updates if said primary volume is updated by said host computer, and a second status in which said storage area is not related to updates if said primary volume is updated by said host computer, a first bitmap, in said third disk array system, storing information related to data received from said first disk array system with said synchronous copy procedure; and a second bitmap, in said third disk array system, storing information related to data that needs to be transmitted from said first disk array system to said second disk array system with said asynchronous remote copy procedure, wherein information stored in said second bitmap are cleared after said second disk array system received data which relates to information stored in said second bitmap from said first disk array system and after said storage area goes through both said first status and said second status, and wherein said third disk array system transmits data which are not received by said second disk array system to said second disk array system, if said first disk array system has a disaster.
 45. A storage system according to claim 44, wherein: said cleared second bitmap is used to newly store information related to data received from said first disk array system with said synchronous copy procedure.
 46. A storage system according to claim 44, wherein: said information stored in said second bitmap are cleared based on one of change between said first status and said second status of said storage area.
 47. A storage system according to claim 44, wherein: said information stored in said second bitmap are cleared, if said storage area changes status which are either said first status or said second status.
 48. A storage system according to claim 44, wherein: said synchronous copy procedure is a copy style that said first disk array system sends acknowledge to said host computer after said third disk array system receives first data sent from said first disk array system, if said first disk array system receives said first data sent from said host computer, and said third disk array system stores information of said first data in said first bitmap, if said third disk array system receives said first data sent from said first disk array system.
 50. A storage system, comprising: a first disk array system coupled to a host computer and controlling to store data sent from said host computer to a first volume, a second disk array system coupled to said first disk array system and receiving data of said first volume from said first disk array system with an asynchronous remote copy procedure and controlling to store data received from said first disk array system to a second volume, a third disk array system coupled to said first disk array system and receiving data of said first volume from said first disk array system with a synchronous copy procedure and controlling to store data received from said first disk array system to a third volume, at least one storage area, in said first disk array system, having a first status in which said storage area is related to updates if said primary volume is updated by said host computer, and a second status in which updates related to said storage area are transferred to said second disk array system, a first bitmap, in said third disk array system, storing information related to data received from said first disk array system with said synchronous copy procedure, data related to said information in said first bitmap corresponding to said updates during a time that said storage area is in said second status; and a second bitmap, in said third disk array system, storing information related to data that needs to be transmitted from said first disk array system to said second disk array system with said asynchronous remote copy procedure, data related to said information in said second bitmap corresponding to said updates related to said storage area of said first status, wherein information stored in said second bitmap are cleared after said second disk array system received data which relates to information stored in said second bitmap, and wherein said first bitmap is stored information, which are used to transfer data from said third disk array system to said second disk array system if said first disk array system has a disaster, when information stored in said second bitmap are cleared.
 51. A storage system, comprising: a first disk array system, in a primary site, coupled to a host computer and having a plurality of first disk drive units in which data are stored and a first controller which controls to store data sent from said host computer to a volume configured by some of said first disk drive units, a second disk array system, in a remote site, coupled to said first disk array system and having a plurality of second disk drive units in which data are stored and a second controller which receives data of said volume from said first disk array system with an asynchronous remote copy and controls to store data received from said first disk array system to said second disk drive units, a third disk array system, in a local site, coupled to said first disk array system and having a plurality of third disk drive units in which data are stored and a third controller which receives data of said volume from said first disk array system with a synchronous copy and controls to store data received from said first disk array system to said third disk drive units, a first area, in said third disk array system, storing information related to data received several times from said first disk array system with said synchronous copy during a first time; a second area, in said third disk array system, storing information related to data received several times from said first disk array system with said synchronous copy during a second time which is a time succeeding said first time, and at least one third area, in said first disk array system, having a first status in which said storage area is related to updates if said primary volume is updated by said host computer, and a second status in which updates related to said third area are transferred to said second disk array system, wherein said first time is a time that said third area is in said first status, wherein said second time is the time that said third area is in said second status, and wherein information stored in said first area and information stored in said second area are cleared at different times so that said third disk array system transfers data, which are not stored by said second disk array system, to said second disk array system after said first disk array system has a failure.
 52. A storage system, comprising: a first disk array system coupled to a host computer and controlling to store data sent from said host computer to a volume, a second disk array system coupled to said first disk array system and receiving data of said volume from said first disk array system with an asynchronous remote copy procedure, a third disk array system coupled to said first disk array system and receiving data of said volume from said first disk array system with a synchronous copy procedure, a first bitmap, in said third disk array system, storing information related to data received from said first disk array system with said synchronous copy procedure during a first time period; a second bitmap, in said third disk array system, storing information related to data received from said first disk array system with said synchronous copy procedure during a second time period which is a time period succeeding said first time period; and at least one area, in said first disk array system, having a first status in which said area is related to updates if said primary volume is updated by said host computer, and a second status in which updates related to said area are transferred to said second disk array system, wherein said first time period is a time period that said area is in said first status, wherein said second time period is the time period that said area is in said second status, wherein said third disk array system clears information stored in said first bitmap after said second disk array system received data which relates to information stored in said first bitmap, and after said first time period, and wherein said third disk array system clears information stored in said second bitmap after said second disk array system received data which relates to information stored in said second bitmap, and after said second time period.
 53. A storage system according to claim 52 wherein: said third disk array system transfers data related to said information stored in said third disk array system during both said first time period and said second time period from said third disk array system to said second disk array system, if said first disk array system has a disaster during transferring data from said first disk array system to said second disk array system with an asynchronous remote copy.
 54. A storage system according to claim 52 wherein: said third disk array system transfers data related to said information stored in said third disk array system during said second time period from said third disk array system to said second disk array system, if said first disk array system has a disaster after transferring data from said first disk array system to said second disk array system with an asynchronous remote copy.
 55. A storage system according to claim 52 wherein: said third disk array system, by using said bitmaps, can transfer lesser amount of data that needs to be transferred to said second disk array system than all data stored in said third volume, if said first disk array system has a disaster.
 56. A storage system according to claim 52, wherein: said third disk array system can minimize amount of data that needs to be transmitted to said second disk array system by using said bitmaps, if said first disk array system has a disaster.
 57. A storage system, comprising: a first disk array system, in a primary site, coupled to a host computer and controlling to store data sent from said host computer to a primary volume, a second disk array system, in a remote site, coupled to said first disk array system and receiving data of said primary volume from said first disk array system with an asynchronous remote copy procedure and controlling to store data received from said first disk array system to a secondary volume, a third disk array system, in a local site, coupled to said first disk array system and receiving data of said primary volume from said first disk array system with a synchronous copy procedure and controlling to store data received from said first disk array system to a third volume, at least one first area, in said first disk array system, having a first status in which said storage area is related to updates if said primary volume is updated by said host computer, and a second status in which said storage area is not related to updates if said primary volume is updated by said host computer, a second area, in said third disk array system, storing information related to data received from said first disk array system with said synchronous copy procedure, data related to said information in said second area corresponding to said updates related to said first area of said first status; and a third area, in said third disk array system, storing information related to data received from said first disk array system with said synchronous copy procedure, data related to said information in said third area corresponding to said updates during a time that said first area is in said second status; wherein said third disk array system clears information stored in said second area after said second disk array system received data which relates to information stored in said second area, and after said first area goes through both said first status and said second status, and wherein information stored in said second area and information stored in said third area are cleared at different times. 